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This study used the Delphi research method to develop the Online Adult Learning Inventory, an instrument 
to apply the principles of adult learning to Web-based instruction. Twelve experts in the fields of adult 
learning and online course development working with the researchers constructed the instrument and 
validated its content. 
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Distance learning is now an important venue where significant adult learning occurs (Brookfield, 1995). “Depending 
on the type of Internet technology a distance course employs, adults will tend to learn differently” and “...the use of 
the Web may require a new commitment to andragogical principles” (Cahoon, 1998, p.29, 34). As a research area 
for consideration. Bates, Holton and Seyler (1996) put forth the challenge to establish normative criteria based on 
adult learning principles (p. 1 8). Course developers need to focus on learning theory in the design of instruction so 
that they can create lessons that they are meaningful and focus on their requirements as an adult (Fidishun, 2000). 

Numerous citations (Cahoon, 1998; Brookfield, 1995; Bates, et al. 1996; Simonson, 1997; Ryan, Carlton, & 
Ali, 1999) reflected the need for further research in computer-mediated instruction for adults and suggested that 
computer design principles for adults may be different (Bates, et al. 1996). Reeves strongly argued that, “...it is 
imperative that criteria for evaluating various forms of CBE (computer-based education) be developed that will 
result in more valid and useful evaluations” (Reeves, 1995. p. 2). He also recommended that any evaluation 
instrument be subject to “rigorous expert review” (p. 11). This challenge and the difficulty in designing a valid 
instrument was met by employing “rigorous expert review” by utilizing experts in the fields of andragogy, 
instructional design, and Web course development to construct the content and structure of the instrument. 

There are some rating systems for Web page style (Jackson, 1998; Waters, 1996; Cyberhound, 1996) and rating 
systems for various applications of adult learning principles (Conti, 1979), measures of self-directed learning 
readiness (Guglielmino, 1992), and Competencies for the Role of Adult Educator/Trainer (Knowles, Holton, & 
Swanson, 1998, p. 140). In addition Wentling and Johnson (1999) developed the Illinois Online Evaluation System 
to judge online instructional efforts in general. Thus, this study’s central problem was that no evaluation instrument 
that specifically deals with the application of adult learning principles (ALR) to Web-based courses and training had 
been identified. Until now, course developers faced a problem because there was no validated list to aid in applying 
adult learning principles to course development or its formative or summative evaluation. The Online Adult 
Learning Inventory (OALI) was developed by the authors and a panel of twelve experts in order to fill that gap. 

The problems and research questions addressed in this study provided the structure, content, and purpose in 
creating an instrument to apply adult learning principles to Web-based instruction and training and included: 

(a) What are examples of specific instructional methods and techniques that demonstrate the application of 
adult learning principles to fully-mediated World Wide Web-based distance education courses or 
training as reported in the literature? 

(b) To what extent can an instrument be developed by a Delphi expert panel to measure the application of 
adult learning principles to fully-mediated World Wide Web-based distance education courses or 
training, either as an ex-post facto evaluation (summative) or as an in-process formative evaluation? 

(c) To what extent is there consensus among Delphi panel experts in the fields of adult education and 
Web-based course development to validate specific instructional methods and techniques that 
demonstrate the application of adult learning principles to fully-mediated World Wide Web-based 
distance education courses or training? 
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The purpose of this study was to develop a validated instrument to help educators, trainers, researchers, and 
instructional designers evaluate and apply the use of adult learning principles to fully- mediated World Wide 
Web-based distance education courses. The theoretical framework of this study was based on a synthesis of 
andragory, instructional design theory, and adult development theory. The instrument constructed in this study 
provides an additional formative and/or summative evaluative tool to assess Web courses or to apply adult learning 
principles to course or training design. The instrument can be printed or downloaded from the following website: 
http://www.mpc.edu/sharon_colton. 

Method 

This study was exploratory in that it relied on qualitative and quantitative consensus-building by a Delphi panel of 
experts to construct and validate content. The content in question was adult learning principles applied to fully- 
mediated World Wide Web-based distance education courses. Research methods for validity included (a) a thorough 
review of the literature to construct an item pool of instructional methods and (b) Delphi expert panel consensus. 
The mean, mode, standard deviation, interquartile range, and skewness of the data were calculated from the voting 
procedures for determination of consensus. Evidence of reliability was indicated by the interrater reliability 
coefficient from a field test. In addition, a review of readability was conducted to improve the readability of the 
instrument and the Gunning Fog Index (1983) for readability was calculated. 

There is a great deal of discussion in the literature concerning the principles of adult learning, particularly those 
principles described by Malcolm Knowles. The literature is rich in evidence of instructional methods for web-based 
courses but far fewer methods that applied principles of adult learning to Web-based instruction. Of those methods, 
some were supported by research and others were developed in the conceptual literature. However, in the literature 
there was no validated list of instructional methods that apply specific adult learning principles to fully-mediated 
World Wide Web courses or training. There was a gap to where the instrument could not be fully constructed just 
from the information in the literature. 

Participants 

The Delphi panel members were rigorously chosen in accordance with established criteria and represented 
excellence in the fields of adult and distance learning as well as instructional design. Each panel member had prior 
working knowledge of adult learning principles and had experience with developing and/or teaching a Web-based 
course or training program, or involvement in distance education programs. Potential panel members were selected 
from the literature based on the number and quality of their publications or experience in the field, particularly 
during the past nine years, a time when Web-based distance learning became feasible. Each potential panel member 
was rated as to their perceived usefulness to the study based on their specific area of expertise. Fifteen potential 
panel members were invited to participate with twelve agreeing to participate. Turoff and Hiltz (1995) suggested ten 
participants to be the minimum. They were asked to sign a consent form prior to participation and give consent for 
their names to be published in the completed research. 

After completion of the Delphi process and an agreed-upon instrument was drafted, a field test was conducted 
to give an indication of the reliability of the instrument. An invitation was sent to all online course developers or 
course evaluators at a West Coast community college to participate in a field test and tutorial on the principles of 
adult learning. Fourteen of the faculty members agreed to participate and signed letters of informed consent. They 
were recruited to use the draft instrument to evaluate a specified instructional Web site. Results of the field test were 
computed to indicate reliability. 

Apparatus 

Computer-based, primarily mainframe -based, Delphi procedures have been used since the 1970s (Turoff & 
Hiltz, 1995). Today, however, the technology is available to conduct an anonymous asynchronous threaded 
discussion easily on the Web “...where the merger of the Delphi process and the computer presents a unique 
opportunity for dealing with situations of unusual complexity” (Turoff & Hiltz, 1995 p.9). Research indicates this 
combination opens the possibility for greater performance from the Delphi panel of experts than could be achieved 
from any individual, something that rarely happens in face-to-face groups (Turoff & Hiltz, 1995, p.8, p.l 1). 

A website was constructed that consisted of a homepage that was referred to as the “Welcome” page, 
assignments, calendar, and threaded discussion forum with attached documents. In addition, the researcher had 
access to a user analysis of the discussion on the Web site. Documents were attached to the discussion forum that 
included draft instruments, text of previous discussions, and voting forms. The welcome page included the 
following internal links: the topic, a short explanation of the Delphi method, and short biographies of the 
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researchers. The voting form when completed by a Delphi expert panel member was automatically e-mailed to the 
researcher. The penname of the expert was included in the voting form. 

Procedure 

The following figure (Figure l:Diagram of the methods) gives a display of the overall methods used in this 
study (Miles & Huberman, 1994). The review of literature, as shown below, provides the structure for, and was key 
to, the remaining research methods. 

Figure 1. Diagram of the Methods. 




The overall research process commenced with a review of the literature. Preliminary content was collected for 
the instrument using established quality filters, criteria for selecting the expert panel were established, and 
appropriate and established research methods were selected. The principles of adult learning were reviewed, as were 
web-based instructional methods with. Selection criteria for panel members were based on a review of the literature, 
potential panel members were selected based on the criteria. 

Members of a mid-western university and college staff were asked to review the preliminary draft instrument 
for appropriate wording and ease of understanding. Revision was made to the wording based on their suggestions. 

Set-up of the discussion forum: The discussion forum was set up on a Web site with the latest revision of the 
instrument and other data attached to the site. Pen names for anonymity and passwords were selected for the 
participants. 
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Round one of the Delphi procedure was the establishment of adult learning principles by discussion and vote for 
possible consensus. The experts were given a draft instrument with adult learning principles, as derived from the 
literature, and were asked if the principles and structure of the instrument were relevant to online learning or needed 
to be revised. They were asked to keep in mind that this list of principles in its final form will serve as the structure of 
the instrument. Prior to voting, the list of adult learning principles was revised based on suggestions by the expert 
panel. Voting ended the round. Results of round one were displayed on the discussion forum. Mean, median, mode, 
standard deviation, and interquartile range were calculated. Based on the suggestions and a statistical analysis of the 
vote, the instrument and its structure and sequence of adult learning principles were again revised. 

Round two of the Delphi was the establishing and sorting of an item pool completed by a vote. Expert panel 
members were asked to list one or more instructional methods that apply to an agreed-upon adult learning principle to 
Web instruction or training for adults. Results of the listing of instructional methods were displayed on the discussion 
forum. Discussion followed and a vote was conducted on the large item pool or list of instructional methods, which 
apply the various adult learning principles to Web courses, using a Likert scale of 1 to 4. (1 - does not apply, 2 - 
moderately applies but not strongly enough to use in the instrument, 3 - applies enough to be included in the 
instrument, and 4 - outstanding application and definitely to include in the instrument). Descriptive statistics were 
calculated, e.g., mean, median, mode, standard deviation, skewness index, interquartile range, and rank to indicate 
consensus. Edits were made by the researcher to the list of instructional methods based on the results of the vote, 
comments on the voting ballot, correspondence, and references from the literature where necessary. 

Round three of the Delphi was a follow up discussion and a second vote on the revised list of instructional items 
either to include in the instrument or consider for elimination. Statistics were calculated as before. Items not having 
reached consensus to be included in the instrument were eliminated from the final instrument. Additional edits were 
made to the list of instructional methods based on the comments of the expert panel. 

A field test was conducted using fourteen community college faculty who had knowledge of Web course 
development and/or evaluation. Comments by the participants related to the draft instrument were recorded. Results 
were analyzed for an indication of inter-rater reliability using standard correlation procedures for estimating 
agreement corrected for chance. The inter-rater reliability statistic gave an indication of the reliability and 
consistency of the instrument. Participant comments and results of the analysis were used for the final revisions of 
the instrument. The Gunning FOG Index (1983) was then computed for an indication of the reading level. 

Results 

Quantitative data were obtained from the voting process of the Delphi expert panel and from the field test of the 
instrument. Qualitative data consisted of theory and excerpts from the literature and over 100 pages of discussion by 
the expert panel members along with additional personal correspondence from individual panel members. 

Table 1 is a summary of the content validity results for the instructional items in each section of the instrument. 
“Mean” is the range of the means calculated for each item in the section. “St Dev” is the range of the standard 
deviations in the section. “IQR” is the interquartile range of each item in the section. A Likert scale of 1 to 4 was 
used (1 - does not apply, 2 - moderately applies but not strongly enough to use in the instrument, 3 - applies enough 
to be included in the instrument, and 4 - outstanding application and definitely to include in the instrument). All final 
content items on the instrument were validated by the expert panel. 



Table 1. Content Validity 



Section 


Mean (range) 


St Dev (range) 


IQR (range) 


Final Status 


Section A 


3.11-3.67 


0.71-1.05 


0-1 


Consensus 


Section B 


3.11-3.78 


0.53-1.05 


0-1 


Consensus 


Section C 


3.11-3.56 


0.73-1.13 


0-1 


Consensus 


Section D 


3.22-3.78 


0.76-1.13 


0-1 


Consensus 


Section E 


3.38-3.50 


0.52-0.74 


1 


Consensus 


Section F 


3.11-3.67 


1.00-1.30 


0-1 


Consensus 


Section G 


3.11-3.89 


0.44-1.13 


0-1 


Consensus 



After the Delphi was complete, a field test was completed with 14 faculty participants who evaluated an online 
(WebCT) college course using the instrument. The average measure intraclass correlation that is essentially the same 
as the Cronbach alpha internal consistency reliability coefficient was computed. The expected range is from zero to 
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1.0. The correlation figures of from .8018 to .9360 indicated moderate to high reliability. The results are 
summarized in the following table (Table 2. Indication of reliability): 



Table 2. Indication of Reliability 



Section 


Average measure intraclass correlation (r n ) 


Section A 


.9360 


Section B 


.8018 


Section C 


.9112 


Section D 


.9112 


Section E 


.9360 


Section F 


.9112 


Section G 


.9360 



To determine the reading level of the instrument, the Gunning FOG Index for each section was calculated as follows 
in Table 3: 



Table 3. Gunning FOG Index 



Section 


Gunning FOG Index (Grade Level) 


Section A 


11.2 


Section B 


11.6 


Section C 


12.2 


Section D 


16.8 


Section E 


12.7 


Section F 


18 


Section G 


11.3 



The reading level or grade levels of items range from high school to graduate school. 

The list of adult learning principles edited for applicability to Web-based courses or training was approved by 
the expert panel. All 43 instructional items in the final instrument received a mean score of 3.11 to 3.89, all with an 
interquartile range of 0 or 1. The criterion for consensus to include an item in the instrument was a mean of 3.0 or 
higher and an interquartile range no greater than 1. All 43 final items met the criteria for consensus. See Table 4 for 
a summary of the results of instructional methods by each ALP. 

The Online Adult Learning Inventory is content valid based on the Delphi techniques summarized here. The 
average measure intraclass correlation results gave moderate to high positive values that communicated that the 
raters were seeing the same thing when they applied the instrument to the distance education course they evaluated, 
an indication of a moderate to high level of reliability. The final instrument as validated by the expert panel is 
available on the following Website in PDF format: http://www.mpc.edu/sharon_colton. 
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Table 4. Tabulation of Instructional Methods by APL 



Adult principle 


Number of methods 
found in literature 


Select examples 


Learner’s need to know 


24 


Orientation session; self-evaluation; record- 
keeping to track progress 


Readiness to learn 


7 


Models; counseling; tasks related to 
developmental stages; 


Self-concept of the learner 


17 


Computer conferences; self-directed learning; 
no competition; share in evaluation; mutual 
inquiry 


Prior experience of the learner 


35 


Group discussion; case method; projects; 
meaningful problems; context of everyday life; 
simulations; peer helping; debates; role playing 


Orientation to learning 


5 


Problem-solving exercises; threaded 

discussions; class calendar 


Motivation to learning 


14 


Activities that promote development of positive 
self-concept; deal with time constraints; 
respectful climate; stimulating tasks; 

enthusiastic atmosphere 


Goals and purposes of learning 


1 


Develop goals during orientation 


Unassigned Web methods 


54 


Create learning community; shared process of 
constructing meaning; telementoring; 

teleapprenticeships; peer tutoring; Delphi 
process for planning and assessment; writing as 
it demand greater reflection than speaking; 
Immediate feedback on quizzes and being 
allowed to take them over again; Advanced 
organizer with a review of the previous lesson 
and a description of the current lesson 



Discussion 

This exploratory study added a validated tool, the Online Adult Learning Inventory, for the evaluation of Web 
courses or training in the workplace to promote excellence in adult learning. Dubois (1997) describes the impact of 
the Information Age on education where “the majority of higher education students will be at least 25 years old and 
where lifelong learning will be ubiquitous” (p. 2). Businesses can also apply this tool to adult training and 
educational courses delivered at a distance by the World Wide Web, a mode that is becoming increasingly common 
(Brown, 1999). To date, no other instruments have been developed specifically for fully-mediated World Wide Web 
courses or training to apply adult learning principles to the instruction. 

Strengths of the Research 

The final design of the instrument, the Online Adult Learning Inventory (http://www.mpc.edu/sharon_colton), 
has both edited principles of adult learning appropriate to online courses and training and practical lists of 
instructional methods that apply the adult learning principles to the development or evaluation of online courses. 
The completed OALI has only seven subscales and 43 instructional items. The following is an example item from 
the OALI: 

D. Because of their prior experiences, adults tend to develop mental habits and biases and may need to 
reassess their beliefs in order to adopt alternate ways of thinking. 

1. Orientation activities are provided at the beginning of the course that allow learners to develop 
the skills necessary to complete the course (e.g., “introduce yourself to the discussion forum,” 
“send me an e-mail saying you were able to log on”). 

The merging of these two constructs offers an innovative and practical tool to address the critical need for online 
learning to adhere to sound adult learning principles. The two parts of the instrument serve secondarily as an 
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educational tool for students, trainers, and educators, as a review of how adults learn differently from traditional 
college age youth. 

Also, the Web-based method was a rigorous and highly innovative approach to instrument development and 
validity that included a threaded discussion forum, and yielded rich data that may not have been garnered through a 
traditional paper-based Delphi process. This may have resulted in a stronger degree of validation by the expert 
panel. In addition, the Delphi technique was deemed the most appropriate method due to the developmental, 
exploratory and contemporary nature of the research. 

Limitations of the Research 

The principle barrier to designing an instrument for measuring adult learning principles in web-based 
environments is the high level of difficulty in establishing its validity and reliability. To overcome this barrier, this 
study utilized experts in andragogy and Web course development to develop the instrument. However, the Delphi 
panel, although recognized experts in andragogy and Web course design, did not include all experts in these fields. 
Also, the field test was conducted on a relatively small sample of the potential audience, thus only an indication of 
reliability could be estimated. 

Implications for HRD Research and Practice 

The Online Adult Learning Inventory, as developed in this study, is new to the field of training, adult learning, 
distance education, and instructional design. Future Web course or training developers can use the instrument to 
construct online learning that is more appropriate to the needs of adult learners and to evaluate and improve the 
online learning environment for their adult learners. It answers the need expressed by Cahoon (1998) in Adult 
Learning and the Internet to develop a checklist for guidelines for web-based course development and evaluation. 
Bates, et al. (1996) put forth the challenge to establish normative criteria based on adult learning principles. Prior to 
this study, no evaluation instrument that specifically dealt with the application of adult learning principles to Web- 
based courses had been identified. The instrument will enable course developers and trainers to apply principles of 
andragogy, or adult learning principles, to the instructional design of a Web-based course. Human resources training 
designers and adult educators can use the instrument to apply the principles of adult learning or andragogy to their 
work in developing instruction or training that meet the learning needs of their adult audiences. For students of 
instructional design or adult education, the instrument also serves as a tutorial in describing the principles of adult 
learning and in selecting instructional methods that apply these principles to Web-based course development. 

The Web-based Delphi process used for this study is also new to the field of research design. This study 
demonstrated the power of technology in enhancing a classic Delphi research process, in facilitating discussion 
among participants separated by time and place, and providing a venue for voting, all while preserving the 
anonymity of the participants. It yielded rich qualitative and rigorous quantitative data resulting in a content 
validated instrument, possibly resulting in a more in-depth content validation, applicable to educational, business, 
industrial, and government research as well as bringing the tenets of andragogy into the 21 st century. 
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